Information processing system, information processing method and program

ABSTRACT

There is provided an information processing apparatus including circuitry configured to control output of a first content according to first content information, determine information related to a real object which is different from a manipulation body used for content selection or object manipulation by a user, and control output of a second content, which is different from the first content, according to second content information, wherein the second content information is based on the first content information and the determined information related to the real object.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/541,576 (filed on Jul. 5, 2017), which is a National Stage PatentApplication of PCT International Patent Application No.PCT/JP2016/000952 (filed on Feb. 23, 2016) under 35 U.S.C. § 371, whichclaims priority to Japanese Patent Application No. 2015-073833 (filed onMar. 31, 2015), the entire contents of which are incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to an information processing system, aninformation processing method and a program.

BACKGROUND ART

Devices configured to display various pieces of information according toa manipulation with respect to a touch panel of a smartphone or a tabletterminal are widespread. In tablet terminals, implementation of a largescreen size and a method of a plurality of users simultaneouslyperforming manipulations are considered. In addition, projectors havebeen used as devices configured to display information since the past.

With the proliferation of such devices configured to displayinformation, various user interfaces for enabling various manipulationswith respect to displayed information have been proposed. For example,in the following Patent Literature 1, a technique for readingcorresponding information such as a URL with reference to a databaseusing an image captured by a mobile terminal as a key is disclosed.

CITATION LIST Patent Literature

[PTL 1]

JP 2006-48672A

SUMMARY Technical Problem

A surrounding environment of a device when information is displayed or astate of displayed information is not always constant. For example,various real objects may be near the device and various display objectsmay be displayed. In view of the above situations, there is a need todisplay information appropriately and efficiently according to the realobject or the display object, and increase user convenience.

Accordingly, the present disclosure proposes a novel and improvedinformation processing system, information processing method and programthrough which it is possible to increase user convenience.

Solution to Problem

According to an aspect of the present disclosure, there is provided aninformation processing apparatus including: circuitry configured tocontrol output of a first content according to first contentinformation, determine information related to a real object which isdifferent from a manipulation body used for content selection or objectmanipulation by a user, and control output of a second content, which isdifferent from the first content, according to second contentinformation, wherein the second content information is based on thefirst content information and the determined information related to thereal object.

According to another aspect of the present disclosure, there is providedan information processing method including: outputting a first contentaccording to first content information, determining information relatedto a real object which is different from a manipulation body used forcontent selection or object manipulation by a user, and outputting asecond content, which is different from the first content, according tosecond content information, wherein the second content information isbased on the first content information and the determined informationrelated to the real object.

According to another aspect of the present disclosure, there is provideda non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute a method, the method including: outputting a first contentaccording to first content information, determining information relatedto a real object which is different from a manipulation body used forcontent selection or object manipulation by a user, and outputting asecond content, which is different from the first content, according tosecond content information, wherein the second content information isbased on the first content information and the determined informationrelated to the real object.

Advantageous Effects of Invention

According to an embodiment of the present disclosure described above, itis possible to increase user convenience. Note that the aboveadvantageous effects are not strictly limiting, and that anyadvantageous effect indicated in the present disclosure or anotheradvantageous effect that may be reasoned from the present disclosure mayalso be exhibited in addition to, or instead of, the above advantageouseffects.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an exterior configuration example of aninformation processing system according to an embodiment.

FIG. 2 is a diagram illustrating an exterior configuration example of aninformation processing system according to an embodiment.

FIG. 3 is a diagram illustrating an exterior configuration example of aninformation processing system according to an embodiment.

FIG. 4 is a block diagram illustrating an exemplary logicalconfiguration of an information processing system according to anembodiment.

FIG. 5 is a block diagram illustrating an exemplary logicalconfiguration of an information processing system according to anembodiment,

FIG. 6 a flowchart illustrating an exemplary flow of correspondencerelation DB update processes performed in an information processingsystem according to an embodiment.

FIG. 7 is a flowchart illustrating an exemplary flow of real objectrecognition processes performed in an information processing systemaccording to an embodiment.

FIG. 8 is a flowchart illustrating an exemplary flow of real objectidentification processes performed in an information processing systemaccording to an embodiment.

FIG. 9 is a diagram for describing an exemplary process of acquiring aposition map according to an embodiment.

FIG. 10 is a diagram for describing an exemplary process of computing aconcordance rate of a position map according to an embodiment.

FIG. 11 is a flowchart illustrating an exemplary flow of real object DBupdate processes performed in an information processing system accordingto an embodiment.

FIG. 12 is a diagram for describing an exemplary process of updating aposition map in a real object DB according to an embodiment.

FIG. 13 is a flowchart illustrating an exemplary flow of providedinformation display processes performed in an information processingsystem according to an embodiment.

FIG. 14 is a flowchart illustrating an exemplary flow of providedinformation display processes performed in an information processingsystem according to an embodiment.

FIG. 15 is a diagram illustrating exemplary information stored in adisplay object DB according to an embodiment.

FIG. 16 is a diagram illustrating exemplary information stored in a realobject DB according to an embodiment.

FIG. 17 is a diagram illustrating exemplary information stored in a realobject DB according to an embodiment.

FIG. 18 is a diagram for describing an exemplary process of identifyinga real object according to an embodiment.

FIG. 19 is a diagram for describing an exemplary process of identifyinga real object according to an embodiment.

FIG. 20 is a diagram for describing an exemplary process of a localcorrespondence relation DB according to an embodiment.

FIG. 21 is a diagram for describing an exemplary process of a localcorrespondence relation DB according to an embodiment.

FIG. 22 is a diagram for describing an exemplary process of a cloudcorrespondence relation DB according to an embodiment.

FIG. 23 is a diagram for describing an exemplary process of a cloudcorrespondence relation DB according to an embodiment.

FIG. 24 is a diagram for describing an exemplary process of a cloudcorrespondence relation DB according to an embodiment.

FIG. 25 is a diagram for describing an exemplary process of a cloudcorrespondence relation DB according to an embodiment.

FIG. 26 is a block diagram illustrating an exemplary hardwareconfiguration of an information processing device according to anembodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described indetail and with reference to the attached drawings. Note that, in thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

Also, in this specification and the appended drawings, elements havingsubstantially the same function and structure may in some cases bedistinguished by different letters appended to the same sign. Forexample, multiple elements having substantially the same function andstructure are distinguished as information processing systems 100A,100B, 100C, and so on as appropriate. On the other hand, when notparticularly distinguishing each of multiple elements havingsubstantially the same function and structure, only the same sign willbe given. For example, the information processing systems 100A, 100B and100C will be simply designated the information processing system 100when not being particularly distinguished.

The description will proceed in the following order.

1. Configuration example

1.1. Exterior configuration example

1.2. Functional configuration example

2. Function details

2.1. DB update function

2.2. Provided information display function

3. Specific examples

4. Hardware configuration example

5. Summary

1. CONFIGURATION EXAMPLE

<1.1. Exterior Configuration Example>

First, an exterior configuration of the information processing system100 according to an embodiment of the present disclosure will bedescribed.

FIG. 1 is a diagram illustrating an exterior configuration example ofthe information processing system 100A according to an embodiment. Asillustrated in FIG. 1, the information processing system 100A accordingto an embodiment includes an input unit 110A and an output unit 130A.The information processing system 100A illustrated in FIG. 1 is a systemthat displays information on a top surface of a table 140A and enables auser who uses the information processing system 100A to manipulate theinformation displayed on the table 140A. A method of displayinginformation on the top surface of the table 140A as illustrated in FIG.1 is referred to as a “projection type.”

The input unit 110A is a device configured to input manipulation contentof the user who uses the information processing system 100A or a shapeor a pattern of an object placed on the table 140A. In the exampleillustrated in FIG. 1, the input unit 110A is provided above the table140A and is suspended from, for example, the ceiling. That is, the inputunit 110A is provided separately from the table 140A serving as a targeton which information is displayed. As the input unit 110A, for example,a camera configured to image the table 140A with one lens or a stereocamera capable of imaging the table 140A with two lenses and recordingdepth direction information (depth information) may be used.

When the camera configured to image the table 140A with one lens is usedas the input unit 110A, if the information processing system 100Aanalyzes an image captured by the camera, it is possible to recognizethe object placed on the table 140A. In addition, when the stereo camerais used as the input unit 110A, the input unit 110A can acquire depthinformation. When the input unit 110A acquires the depth information,the information processing system 100A can recognize, for example, ahand or the object placed on the table 140A. In addition, when the inputunit 110A acquires the depth information, the information processingsystem 100A can recognize contact, approach or release between theuser's hand and the table 140A. Note that, in the following description,a case in which the user brings a manipulation body such as a hand incontact with or close to a display surface of information is simplyreferred to in general as “contact.” In this manner, the input unit 110Afunctions as an image sensor configured to image the table 140A, a depthsensor configured to acquire depth information on the table 140A and atouch sensor configured to recognize contact on the table 140A.

Hereinafter, a case in which the user's manipulation is recognized froman image captured by the input unit 110A will be mainly described, butthe present disclosure is not limited thereto. The user's manipulationmay also be recognized by a touch panel configured to recognize contactof the user's finger or the like. Also, the user's manipulation that theinput unit 110A can acquire may include, for example, a stylusmanipulation with respect to the display surface of information and agesture manipulation with respect to a camera, in addition thereto.

The output unit 130A is a device that displays information on the table140A and outputs audio according to manipulation content of the user whouses the information processing system 100A, which is input by the inputunit 110A, content of information output from the output unit 130A, andinformation of a shape, a pattern or the like of the object placed onthe table 140A. As the output unit 130A, for example, a projector or aspeaker is used. In the example illustrated in FIG. 1, the output unit130A is provided above the table 140A and is suspended from, forexample, the ceiling. When the output unit 130A is formed by theprojector, the output unit 130A projects information on the top surfaceof the table 140A.

The user who uses the information processing system 100A can place afinger or the like on the table 140A and perform a manipulation withrespect to information on the table 140A displayed by the output unit130A. In addition, when the object is placed on the table 140A andrecognized by the input unit 110A, the user who uses the informationprocessing system 100A can perform various manipulations for therecognized object.

Also, although not illustrated in FIG. 1, other devices may be connectedto the information processing system 100A. For example, a lightingdevice for illuminating the table 140A may be connected to theinformation processing system 100A. When the lighting device forilluminating the table 140A is connected to the information processingsystem 100A, the information processing system 100A can control alighting state of the lighting device according to a state of thedisplay surface of information.

A mode of the information processing system 100 according to anembodiment of the present disclosure is not limited to the illustrationin FIG. 1. Hereinafter, other modes of the information processing system100 according to an embodiment will be exemplified with reference toFIGS. 2 and 3.

FIG. 2 is a diagram illustrating an exterior configuration example ofthe information processing system 100B according to an embodiment. Inthe information processing system 100B illustrated in FIG. 2, a touchpanel type display is formed on a top surface of a table 140B. In thepresent configuration example, the output unit 130B may be formed by thetouch panel type display. That is, in the present configuration example,the display surface of information is the touch panel type display. Amethod of displaying information on the touch panel provided on the topsurface of the table 140B as illustrated in FIG. 2 is also referred toas a “touch panel type.” As illustrated in FIG. 2, the input unit 110Bmay be formed by the touch panel type display or may be provided abovethe table 140B and separated from the table 140B. For example, the inputunit 110B formed by the touch panel type display serves as the touchsensor, and the input unit 110B provided above the table 140B serves asthe image sensor and the depth sensor.

FIG. 3 is a diagram illustrating an exterior configuration example ofthe information processing system 100C according to an embodiment. Theinformation processing system 100C illustrated in FIG. 3 is formed suchthat information is projected to the output unit 130C from below a table140C, and thus the information is displayed on a surface of the table140C. That is, in the information processing system 100C illustrated inFIG. 3, the display surface of information is a top surface of the table140C. A surface of the table 140C is formed of a transparent orsemitransparent material such as a glass plate or a transparent plasticplate. A method in which information is projected to the output unit130C from below the table 140C and thus the information is displayed onthe top surface of the table 140C as illustrated in FIG. 3 is alsoreferred to as a “rear projection type.” As illustrated in FIG. 3, aninput unit 110C may be provided above the table 140C and separated fromthe table 140C, or may be provided below the table 140C and separatedfrom the table 140C. For example, the input unit 110C provided above thetable 140C serves as the image sensor and the depth sensor, and theinput unit 110C provided below the table 140C serves as the touchsensor.

In the following description, the projection type information processingsystem 100 illustrated in FIG. 1 will be exemplified. In addition,information displayed by the output unit 130 such as an applicationwindow, an image or a video is also referred to as “a display object.”Hereinafter, the display object will be described as the applicationwindow. In addition, the object which is placed on or held over the topsurface of the table 140A near the information processing system 100 isalso referred to as “a real object.” The user's body may also beregarded as the real object. The display object may be displayed onvarious object surfaces. For example, the display object may beprojected on the top surface of the table 140A, or may be projected onthe real object such as a screen placed on the table 140A. In thisspecification, the object surface on which the display object isdisplayed is also simply referred to as a “display surface.”

<1.2. Functional Configuration Example>

Next, a functional configuration of the information processing system100 according to an embodiment will be described.

FIG. 4 is a block diagram illustrating an exemplary logicalconfiguration of the information processing system 100 according to anembodiment. As illustrated in FIG. 4, the information processing system100 according to an embodiment includes the input unit 110, a controlunit 120, the output unit 130 and a storage unit 150.

(1) Input Unit 110

The input unit 110 has a function of receiving an input of variouspieces of information to the input unit 110. For example, the input unit110 functions as the image sensor configured to image the table 140A,the depth sensor configured to acquire depth information on the table140A and the touch sensor configured to recognize contact on the table140A as described above. A range in which information is displayed bythe output unit 130 and a range in which the input unit 110 receives aninput of information may be the same or different. For example, theinput unit 110 may acquire image data, depth information, or contactinformation from a wider range than the range displayed by the outputunit 130.

As the image sensor, the input unit 110 may be formed by, for example, avisible light camera or an infrared camera. As the depth sensor, theinput unit 110 may be formed by, for example, a stereo camera, adistance measuring sensor using a time of flight method, or a distancemeasuring sensor using a structured light method. As the touch sensor,the input unit 110 may also be formed by a touch panel, or may recognizecontact according to depth information.

In addition, the input unit 110 may be formed by a biometric sensorconfigured to acquire biometric information such as a body temperature,perspiration, or a pulse of the user.

(2) Control Unit 120

The control unit 120 serves as an arithmetic processing unit and acontrol device, and controls overall operations in the informationprocessing system 100 according to various programs. For example, thecontrol unit 120 uses information input by the input unit 110 andgenerates information to be output from the output unit 130. Asillustrated in FIG. 4, the control unit 120 serves as a recognition unit122, a storage control unit 124 and a display control unit 126.

(2.1) Recognition Unit 122

The recognition unit 122 has a function of recognizing the real objectnear the information processing system 100. For example, the recognitionunit 122 recognizes the display object and the real object on the object(for example, the table 140A illustrated in FIG. 1) surface on which thedisplay object is displayed. For example, the recognition unit 122 mayrefer to information output by the output unit 130 and recognize thedisplay object that is displayed on the display surface. In addition,the recognition unit 122 may recognize a shape, a pattern or the like ofthe real object based on image information and depth information inputby the input unit 110. In addition, the recognition unit 122 mayrecognize the user's biometric information based on a sensing resultinput by the input unit 110. In addition, the recognition unit 122 mayrecognize a nearby device such as a wristwatch worn by the user or agripped smartphone based on image information and depth informationinput by the input unit 110.

Also, when the information processing system 100 has the projection typeillustrated in FIG. 1, coordinates of the display surface of informationand coordinates of the manipulation body such as the user's hand incontact with the display surface are calibrated in advance to match.Therefore, the recognition unit 122 can recognize that the manipulationbody such as the user's hand comes in contact with a part of the GUI.

(2.2) Storage Control Unit 124

The storage control unit 124 has a function of accessing the storageunit 150 and acquiring and storing information.

For example, the storage control unit 124 may cause informationassociated with a combination of the display object and the real objectrecognized by the recognition unit 122 to be stored in the storage unit150. Therefore, at least one display object displayed on the displaysurface and at least one real object on the display surface which arerecognized at the same timing are associated. Hereinafter, informationindicating such a correspondence relation is also referred to as“correspondence relation information.” In addition, the storage controlunit 124 may acquire the correspondence relation information between thedisplay object and the real object from the storage unit 150. Therefore,the storage control unit 124 can acquire information indicating acombination of the display object and the real object which werepreviously recognized at the same timing.

The storage unit 150 is a unit configured to record data in apredetermined recording medium and reproduce the data. The storage unit150 may store a display object DB (database), a real object DB, and acorrespondence relation DB, which will be described below. The storageunit 150 may store specific information in the information processingsystem 100, or may store information shared with another informationprocessing system 100. For example, the storage unit 150 may be providedin the information processing system 100 and may be provided as aseparate device such as a server connected to a network or the like. Forconvenience of description, in this specification, specific informationin the information processing system 100 may be described as “local” andinformation shared with another information processing system 100 may bedescribed with “cloud.”

The display object DB stores information on the display object. Forexample, the display object DB stores information shown in the followingTable 1. Identification information is information for identifying thedisplay object. The identification information may include informationfor uniquely identifying the display object itself and may includeinformation indicating a category of the display object such as an“entertainment application” when the display object is, for example, amusic player. Content of the display object is information indicatingcontent of the display object such as an application type.

TABLE 1 Identification information Content of display object . . . . . .

The real object DB stores information on the real object. For example,the real object DB stores information shown in the following Table 2.Identification information is information for identifying the realobject. The identification information may include information foruniquely identifying the real object itself, and may include informationindicating a category of the real object such as “tableware” when thereal object is, for example, a coffee cup. Image information isinformation indicating a captured image of the real object obtained byimage data acquired by the input unit 110. 3D shape information isinformation indicating a 3D shape of the real object, and is, forexample, depth information of the real object acquired by the input unit110. A position map is information indicating a probability of thepresence of the real object in the display surface obtained by the depthinformation acquired by the input unit 110. Other attribute informationis information indicating attributes of the real object, and mayinclude, for example, information indicating a time period in which thereal object is recognized or information indicating a temperature of thereal object. Also, the information indicating a time period may be anyunit such as units of minutes, hours, or days, or days may bedistinguished by days of the week or as weekdays/holidays.

TABLE 2 Identification Image 3D shape Position Other attributeinformation information information mapinformation - - - - - - - - - - - - - - -

The correspondence relation DB stores the correspondence relationinformation. For example, the correspondence relation DB storesinformation shown in the following Table 3. Identification informationof an entry is identification information of the correspondence relationinformation. Identification information of the display object isidentification information for identifying the associated displayobject, and may include at least one piece of identificationinformation. Identification information of the real object isidentification information for identifying the associated real objectand may include at least one piece of identification information.Arrangement information is information indicating a relative arrangementbetween the real object and the display object. Information indicatingan arrangement includes a relative direction, distance, orientation,size or the like. A degree of association is information indicating adegree of association between the real object and the display object,and increases as, for example, a time for which the display object andthe real object are simultaneously used or the number of times thereofincreases. In addition, as the correspondence relation information,information indicating a time lag may be included. For example,information indicating a time lag from when the user places the realobject on the table 140A until the display object is displayed or a timelag opposite thereto may be included.

TABLE 3 Identification Identification Identification informationinformation information of of of Arrangement Degree of entry displayobject real object information association - - - - - - - - - - - - - - -

Also, while content of the DB has been shown above in the table, it maybe implemented in the form of, for example, a relational database.

(2.3) Display Control Unit 126

The display control unit 126 has a function of controlling the outputunit 130 to cause information to be output.

For example, the display control unit 126 generates a display controlsignal for controlling display of the display object according toinformation input by the input unit 110. Specifically, the displaycontrol unit 126 performs drawing control of the display object orscreen transition control according to a touch on each display object.

For example, the display control unit 126 may cause provided informationcorresponding to at least any of a recognized display object and realobject to be displayed on the output unit 130 based on a recognitionresult by the recognition unit 122. Therefore, the user may receive, forexample, the provision of information of the real object correspondingto the display object that is displayed, and receive the provision ofinformation of the display object corresponding to the real objectplaced on the table 140A. In this case, the display control unit 126 mayalso display the provided information based on the correspondencerelation information acquired by the storage control unit 124. In thiscase, the user can receive the provision of information including her orhis own use method of the display object and the real object in the pastand content according to use methods of other users. In this manner, theuser convenience may be increased.

A display form and content of the provided information may be changedaccording to a recognition result by the recognition unit 122. First,the display form will be described.

For example, the display control unit 126 may control an arrangementsuch as a position, a size and/or an orientation in which the providedinformation is displayed based on identification information of therecognized display object and/or real object. Specifically, in anenvironment in which a coffee cup (the real object) is put down and aweb browser (the display object) is displayed, the display control unit126 may employ an arrangement corresponding to the coffee cup, anarrangement corresponding to the web browser, or an arrangementcorresponding to the coffee cup and the web browser. In addition, in thesame environment, the display control unit 126 may employ an arrangementcorresponding to a category “tableware,” an arrangement corresponding toa category “entertainment applications,” or an arrangement correspondingto the category “tableware” and the category “entertainmentapplications.”

In addition, the display control unit 126 may cause the providedinformation to be displayed based on a position of the recognizeddisplay object and/or real object. Specifically, the display controlunit 126 may control an arrangement of the provided information based onthe position of the recognized display object and/or real object.Further, the display control unit 126 may control a display parametersuch as brightness or contrast. Therefore, the user can receive theprovision of information in the display form according to a position atwhich the real object is placed and/or a position at which the displayobject is displayed.

The display control unit 126 may control an arrangement based on variouspieces of information in addition to the position of the display objectand the real object. For example, the display control unit 126 maycontrol an arrangement based on at least any piece of information shownin the following example.

-   -   Position, size and/or orientation of real object    -   Position, size and/or orientation of display object    -   Relative position and/or orientation between real object and        display object    -   State of process of display object    -   Time period    -   User profile

According to the information exemplified above, even if the same realobject and/or display object are recognized, the display control unit126 may employ a different arrangement when an absolute position, size,and/or orientation are different. In addition, even if the same realobject and/or display object are recognized, the display control unit126 may employ a different arrangement when a relative position and/ororientation are different. In addition, even if the same display objectis recognized, the display control unit 126 may employ a differentarrangement according to a state of a process of the display object suchas that a process is in progress or a user input is being awaited. Forexample, the display control unit 126 may cause the provided informationto be displayed in a great scale when an application awaits a userinput, and may cause the provided information to be displayed in a smallscale so as not to interfere when the user input is in progress. Inaddition, even if the same real object and/or display object arerecognized, the display control unit 126 may employ a differentarrangement when a time period and/or a user profile are different.

The display form has been described above. Next, content will bedescribed.

For example, the provided information may be generated and displayedbased on identification information of the recognized display objectand/or real object. Specifically, in an environment in which the coffeecup is put down and the web browser is displayed, the display controlunit 126 may cause provided information corresponding to the coffee cupto be displayed, provided information corresponding to the web browserto be displayed, and provided information corresponding to the coffeecup and the web browser to be displayed. In addition, in the sameenvironment, the display control unit 126 may cause provided informationcorresponding to the category “tableware” to be displayed, providedinformation corresponding to the category “entertainment applications”to be displayed, and provided information corresponding to the category“tableware” and the category “entertainment applications” to bedisplayed.

Here, an example of provided information corresponding to the displayobject and the real object will be described. For example, in anenvironment in which the coffee cup is put down and the web browser isdisplayed, a music player may be displayed as the provided informationcorresponding to the coffee cup and the web browser. In addition, in anenvironment in which the coffee cup is put down and a news applicationis displayed, a schedule application may be displayed as providedinformation corresponding to the coffee cup and the news application. Inaddition, in an environment in which a magazine is put down and the newsapplication is displayed, an advertisement may be displayed as providedinformation corresponding to the magazine and the news application.

In addition, the provided information may be generated based on theposition of the recognized display object and/or real object.Specifically, the correspondence relation information according to acombination of the same display object and real object is handled asdifferent correspondence relation information when arrangementinformation is different. Therefore, when the position of the recognizeddisplay object and/or real object is different, in order to generateprovided information, different correspondence relation information mayhe referred to, and different provided information may be generated.Therefore, the user can receive the provision of informationcorresponding to the position at which the real object is placed and/orthe position at which the display object is displayed.

The provided information may be generated based on various pieces ofinformation in addition to the position of the display object and thereal object. For example, the display control unit 126 may cause theprovided information to be displayed based on at least any piece ofinformation exemplified below,

-   -   Position, size and/or orientation of real object    -   Position, size and/or orientation of display object    -   Relative position and/or orientation between real object and        display object    -   State of process of display object    -   Time period    -   User profile

According to the information exemplified above, even if the same realobject and/or display object are recognized, the display control unit126 may cause different provided information to be displayed when anabsolute position, size, and/or orientation are different. In addition,even if the same real object and/or display object are recognized, thedisplay control unit 126 may cause different provided information to bedisplayed when a relative position and/or orientation are different. Inaddition, even if the same display object is recognized, the displaycontrol unit 126 may cause different provided information to bedisplayed according to a state of a process of the display object. Forexample, the display control unit 126 may cause a music player to bedisplayed in order to effectively utilize a waiting time when the webbrowser is downloading a file, and cause a file manager to be displayedafter downloading is completed. In addition, even if the same realobject and/or display object are recognized, the display control unit126 may cause different provided information to be displayed when a timeperiod and/or a user profile are different.

The display form and content of the provided information which arechanged according to the recognition result have been described above.

When a part of the combination of the correspondence relationinformation is recognized and the other part thereof is not recognized,the display control unit 126 may display information on the displayobject or the real object which is not recognized as the providedinformation. The provided information may also be information of thedisplay object associated with the real object recognized by therecognition unit 122 in the correspondence relation information. Forexample, the display control unit 126 may also display the displayobject when the display object corresponding to the recognized realobject is not yet displayed. Therefore, for example, when the userplaces only the real object on the table 140A, an application that isoften used together with the real object is displayed. The providedinformation may also be information of the display object associatedwith the display object recognized by the recognition unit 122 in thecorrespondence relation information. For example, when the real objectcorresponding to the display object that is displayed is not yetrecognized, the display control unit 126 may also display the displayobject related to the real object. Therefore, when the user activates,for example, only an application, the display object serving as areminder of the real object which is often used together with theapplication is displayed.

Alternatively, the provided information may also be advertisementinformation of the display object or the real object. For example, thedisplay control unit 126 may also display advertisement informationassociated with at least any of the display object and the real objectrecognized by the recognition unit 122. Therefore, the user can receive,for example, introduction of the activated application or introductionof a product and an application associated with the real object placedon the table 140A. For example, when the user places the coffee cup,chocolates or cookies may be recommended. The advertisement informationmay be registered in the cloud correspondence relation DB by, forexample, a third party such as a company.

The display control unit 126 may also display the provided informationbased on the correspondence relation information obtained by theinformation processing system 100 itself. That is, the display controlunit 126 may also display the provided information with reference to alocal correspondence relation DB. Therefore, the user can receive theprovision of information according to her or his own use method.Alternatively, the display control unit 126 may also display theprovided information based on the correspondence relation information inanother information processing system 100. That is, the display controlunit 126 may also display the provided information with reference to thecloud correspondence relation DB. Therefore, the user can receive theprovision of information based on integrated knowledge including otherusers who use the same use method as the user. In addition, for example,when the information processing system 100 is installed in eachdifferent room in one house, in consideration of a case in which thereis a difference between uses of the display object and the real object,the mutual correspondence relation DB may be referred to. Further, thedisplay control unit 126 may also display the provided information basedon the correspondence relation information of another user who issimilar to the user recognized by the recognition unit 122. For example,the storage control unit 124 refers to pre-registered user profileinformation, determines a sex or an age group from a hand image or avoice, determines whether she or he is married based whether or not sheor he is wearing a ring, or determines a race according to a color or aregion of skin. Therefore, the display control unit 126 may also displaythe provided information based on the correspondence relationinformation registered in the cloud correspondence relation DB relatedto another user who has a similar profile to the user. Therefore, theuser can receive the provision of information based on integratedknowledge related to another user who has a similar profile to the user.

The control unit 120 may be formed by, for example, a central processingunit (CPU). When the control unit 120 is formed by a device such as theCPU, such a device may be formed by an electronic circuit.

In addition, although not illustrated in FIG. 4, the control unit 120may have a communication function of wirelessly communicating with otherdevices or a function of controlling operations of other devicesconnected to the information processing system 100, for example, thelighting device.

(3) Output Unit 130

The output unit 130 has a function of outputting information based oncontrol according to the display control unit 126. For example, theoutput unit 130 serves as a display unit configured to display thedisplay object according to a display control signal generated by thedisplay control unit 126. The output unit 130 may be formed by, forexample, a touch panel, a liquid crystal screen or a projector.

The information processing system 100 illustrated in FIG. 4 may beformed by a single device, or a part or all thereof may be formed byseparate devices. For example, in the functional configuration exampleof the information processing system 100 illustrated in FIG. 4, thecontrol unit 120 may be provided in a device such as a server that isconnected to the input unit 110 and the output unit 130 via a network.When the control unit 120 is provided in the device such as the server,information from the input unit 110 is transmitted to the device such asthe server via the network, the control unit 120 performs a processaccording to the information from the input unit 110, and information tobe output by the output unit 130 is transmitted to the output unit 130from the device such as the server via the network. Such a configurationexample will be described below with reference to FIG. 5.

FIG. 5 is a block diagram illustrating an exemplary logicalconfiguration of the information processing system 100 according to anembodiment. As illustrated in FIG. 5, the information processing system100 according to an embodiment includes an input and output device 200,a server 300 and the storage unit 150.

The input and output device 200 includes a communication unit 160 andthe above-described input unit 110 and output unit 130. Thecommunication unit 160 has a function of communicating with the server300. The communication unit 160 transmits information input by the inputunit 110 to the server 300, and outputs the display control signalreceived from the server 300 to the output unit 130.

The server 300 includes a communication unit 170, and theabove-described recognition unit 122, storage control unit 124 anddisplay control unit 126. The communication unit 170 has a function ofcommunicating with the input and output device 200. The communicationunit 170 receives information input by the input unit 110 from the inputand output device 200, and transmits the display control signal to theinput and output device 200.

The functional configuration example of the information processingsystem 100 according to an embodiment has been described above. Next,functions of the information processing system 100 according to anembodiment will be described in detail with reference to FIGS. 6 to 14.

2. FUNCTION DETAILS

<2.1. DB Update Function>

First, the DB update function will be described. The informationprocessing system 100 updates the correspondence relation DB, the realobject DB and the display object DB as a prior preparation fordisplaying the provided information. Hereinafter, such an update processwill be described. Note that, in the display object DB, since thedisplay object that may be displayed is simply registered or updated inthe DB, description thereof will be omitted.

(Correspondence Relation DB Update Process)

FIG. 6 is a flowchart illustrating an exemplary flow of correspondencerelation DB update processes performed in the information processingsystem 100 according to an embodiment.

As illustrated in FIG. 6, first, in step S102, the recognition unit 122determines whether an available display object is displayed. Therecognition unit 122 determines that, for example, an application thatis iconized and not used by the user is not available. In addition, therecognition unit 122 determines that, for example, an application whosetime since activation is less than a predetermined time is notavailable. In this manner, the following process is prevented from beingperformed on the display object that is not used or the display objectthat is incorrectly displayed. When it is determined that the availabledisplay object is not displayed (NO in S102), the process returns tostep S102 again.

When it is determined that the available display object is displayed(YES in S102), in step S104, the information processing system 100performs a real object recognition process. The real object recognitionprocess is a process for recognizing the real object. This process willbe described below in detail with reference to FIG. 7. Within theavailable display object, using the display object on which processes ofsteps S106 to S122 are not performed as a target, processes after thisstep are performed for each target display object.

Next, in step S106, the recognition unit 122 determines whether the realobject is recognized in step S104. When it is determined that the realobject is not recognized (NO in S106), the process advances to thefollowing step S124.

When it is determined that the real object is recognized (YES in S106),in step S108, the information processing system 100 performs a realobject identification process. The real object identification process isa process of identifying the target real object as the real objectregistered in the real object DB. This process will be described belowin detail with reference to FIG. 8. Within the recognized real object,using the real object on which processes of steps S108 to S120 are notperformed as a target, processes after this step are performed for eachtarget real object.

Next, in step S110, the information processing system 100 performs areal object DB update process. The real object DB update process is aprocess of updating the real object DB. This process will be describedbelow in detail with reference to FIG. 11.

Next, in step S112, the storage control unit 124 determines whether thecorrespondence relation information of a combination of the availabledisplay object and the target real object is registered in thecorrespondence relation DB.

When it is determined that the information is not registered (NO inS112), in step S120, the storage control unit 124 registers an entry inthe local correspondence relation DB. For example, the storage controlunit 124 adds an entry of the correspondence relation information to thecorrespondence relation DB. In this case, the storage control unit 124registers arrangement information based on an arrangement relationbetween the real object and the display object, for example, adirection, a distance, an orientation or a size of the display objectseen from the real object. In addition, the storage control unit 124sets an initial value of a degree of association. Then, the processadvances to the following step S122.

When it is determined that the information is registered (YES in S112),in step S114, the storage control unit 124 updates the entry of thelocal correspondence relation DB. For example, the storage control unit124 increases a degree of association of the correspondence relationinformation in the correspondence relation DB. Therefore, a probabilityof the provided information being provided based on the correspondencerelation information increases. Also, when a direct manipulation withrespect to the correspondence relation DB or the provided information isrejected by the user, the degree of association may decrease. Therefore,a probability of the provided information being provided based on thecorrespondence relation information decreases. In addition, the storagecontrol unit 124 may update arrangement information of the entryaccording to the arrangement relation between the real object and thedisplay object.

Next, in step S116, the storage control unit 124 determines whether theupdated entry satisfies predetermined conditions. For example, when adegree of association exceeds a threshold value, the storage controlunit 124 determines that predetermined conditions are satisfied, andotherwise, the storage control unit 124 determines that predeterminedconditions are not satisfied. When it is determined that thepredetermined conditions are satisfied (YES in S116), in step S118, thestorage control unit 124 updates a cloud information provision DB. Whenthe entry is not registered, the storage control unit 124 registers theentry. When the entry is registered, the storage control unit 121updates the entry. Therefore, information of an entry having a highdegree of association is shared with another information processingsystem 100. When it is determined that predetermined conditions are notsatisfied (NO in S116), the process advances to the following step S122.

In step S122, the storage control unit 124 determines whether all targetreal objects are processed. When it is determined that there is anunprocessed real object (NO in S122), the process returns to step S108again. When it is determined that the objects are processed (YES inS122), in step S124, the recognition unit 122 determines whether allavailable display objects are processed. When it is determined that theobjects are processed (YES in S124), the process ends. Otherwise, whenit is determined that there is an unprocessed display object (NO inS124), the process returns to step S104 again.

(Real Object Recognition Process)

FIG. 7 is a flowchart illustrating an exemplary flow of real objectrecognition processes performed in the information processing system 100according to an embodiment.

As illustrated in FIG. 7, first, in step S202, the recognition unit 122sets depth information of an initial state. For example, the recognitionunit 122 sets depth information on the display surface at a certain timeas the depth information of an initial state. The recognition unit 122may set a state of a time at which the information processing system 100is activated as the depth information of an initial state, or may updatethe depth information of an initial state whenever the real object isadded or removed. The depth information of an initial state may be depthinformation of a state in which there is no real object on the displaysurface or may be depth information when any real object is on thedisplay surface.

Next, in step S204, the recognition unit 122 acquires depth informationon the display surface. Therefore, in step S206, the recognition unit122 compares the depth information of an initial state with the acquireddepth information. For example, when there is a region having a higherheight than that of the depth information of an initial state in theacquired depth information, the recognition unit 122 determines that thereal object is added to the region. Otherwise, when there is a regionhaving a lower height than that of the depth information of an initialstate in the acquired depth information, the recognition unit 122determines that the real object is removed from the region.

When it is determined that the real object is added (YES in S208), instep S210, the recognition unit 122 recognizes the added real object asthe target real object. According to this step, it is possible toprevent an immovable real object such as a tissue box constantly placedon the table 140A, an irregularity of the table 140A or the like frombeing handled as the target.

On the other hand, when it is determined that the real object is notadded (NO in S208), if it is determined that the real object is removed(YES in S212), in step S214, the recognition unit 122 excludes theremoved real object from a recognition target.

In this manner, the recognition unit 122 continuously monitors whetherthe real object is added to or removed from the display surface.Therefore, the information processing system 100 can appropriatelyrecognize the real object that is put down or removed by the user at anytime.

(Real Object Identification Process)

FIG. 8 is a flowchart illustrating an exemplary flow of real objectidentification processes performed in the information processing system100 according to an embodiment.

As illustrated in FIG. 8, first, in step S302, the recognition unit 122acquires information of the target real object. For example, therecognition unit 122 acquires a captured image, information indicating a3D shape and information indicating a position at the display surface ofthe target real object. Specifically, the recognition unit 122 acquiresa captured image, depth information, and a position map of the realobject input by the input unit 110. Here, a method of acquiring aposition map will be described with reference to FIG. 9.

FIG. 9 is a diagram for describing an exemplary process of acquiring aposition map according to an embodiment. As illustrated in FIG. 9, acoffee cup 301 is placed on the top surface of the table 140A by theuser. Then, the recognition unit 122 recognizes a region 302 having aheight based on the depth information, and acquires a position map 303in which a range of the region 302 in a coordinate system of the topsurface of the table 140A is represented as a region in which the realobject is located. In the position map 303, a presence probability is 1in a grid in which a height is recognized and is 0 in the other grids.In FIG. 9, grids are expressed in different shades according to a levelof the presence probability, and the presence probability increases asthe grids darken. The coordinate system of the position map is, forexample, an orthogonal coordinate system in which a top surface on thetable 140A is divided into grids in units of 5 cm.

Next, in step S304, the storage control unit 124 acquires an entry ofthe real object DB. For example, the storage control unit 124 acquiresall entries registered in the real object DB.

Then, in step S306, the storage control unit 124 identifies the realobject. For example, the storage control unit 124 identifies the realobject according to at least any of a captured image, informationindicating a 3D shape and information indicating a position at theobject surface of the real object. Specifically, the storage controlunit 124 searches the real object DB for an entry (image information, 3Dshape information and a position map) similar to at least any of acaptured image, depth information, and a position map of the real objectacquired in step S302. For example, the storage control unit 124compares the target real object with the real object registered in theentry and calculates an evaluation value based on the followingequation, and identifies the real object based on the evaluation value.

Score=F _(i) M+F _(d) M _(d) +F _(p) M _(p)   (Equation 1)

Here, F_(i), F_(d) and F_(p) are weight coefficients whose sum is 1. Thestorage control unit 124 can control the weight coefficient according toan environment or settings. M_(i) denotes a concordance rate of thecaptured image. The concordance rate corresponds to a similarity rate ina general object recognition technique using an image. M_(d) denotes aconcordance rate of depth information. According to M_(i), it ispossible to identify a real object having no texture and real objectshaving the same appearance and different heights that could not herecognized using only an image. M_(i) may be computed in the same way asa method of computing a concordance rate of a position map, which willbe described below, M_(p) denotes a concordance rate of a position map.According to M_(p) an image and a 3D shape match. That is, it ispossible to identify a real object having the same appearance andheight. Here, a method of computing M_(p) will be described withreference to FIG. 10.

FIG. 10 is a diagram for describing an exemplary process of computing aconcordance rate of a position map according to an embodiment. Thereference numeral 311 indicates a position map of the real objectregistered in the real object DB. The reference numeral 312 indicates aposition map of the real object acquired in step S302. For example, thestorage control unit 124 obtains a position map 313 having an absolutevalue of a difference between a position map 311 and a position map 312and computes an average of differences in respective grids of theposition map 313 as the concordance rate of the position map.

As an example, an example in which the recognized real objects A to Care identified as a registered real object will be described. Forexample, when M_(i), M_(d) and M_(p) are computed as shown in thefollowing Table 4, evaluation values of the real objects A to C arecomputed as in the following equation. The storage control unit 124 mayidentify the real object A having the greatest evaluation value as theregistered real object.

F _(i)=0.5, F _(d)=0.3, F _(p) =0.2

Score_(A)=0.5×0.9+0.3×0.9+0.2×0.9=0.9

Score_(B)=0.5×0.9+0.3×0.1+0.2×0.8=0.64

Score_(C)=0.5×0.6+0.3×0.2+0.2×0.9=0.54   (Equation 2)

TABLE 4 Real object identification information M_(i) M_(d) M_(p) A 90%90% 90% B 90% 10% 80% C 60% 20% 90%

The storage control unit 124 may also identify the real object furtherbased on other attribute information. For example, the storage controlunit 124 may identify the real object according to informationindicating a time period in which the real object is recognized.Therefore, the storage control unit 124 can identify the same realobject as a different real object when the time period (including atime, a day of the week, a weekday/a holiday) is different. In addition,the storage control unit 124 may identify the real object according to,for example, a temperature of the real object. Therefore, the storagecontrol unit 124 can identify the same real object as a different realobject when the temperature is different.

(Real Object DB Update Process)

FIG. 11 is a flowchart illustrating an exemplary flow of real object DBupdate processes performed in the information processing system 100according to an embodiment.

As illustrated in FIG. 11, first, in step S402, the storage control unit124 determines whether the target real object is registered in the realobject DB. For example, the storage control unit 124 determines whetherthe target real object is identified as the real object registered inthe real object DB according to the real object identification processdescribed above with reference to FIG. 8.

When it is determined that the object is registered (YES in S402), instep S404, the storage control unit 124 updates the entry of the realobject DB. For example, the storage control unit 124 updates theposition map of the entry of the real object DB. Here, a method ofupdating the position map will be described with reference to FIG. 12.

FIG. 12 is a diagram for describing an exemplary process of updating aposition map in the real object DB according to an embodiment. In FIG.12, the reference numeral 321 indicates a position map of the realobject recognized this time and the reference numeral 322 indicatesposition maps of the real object the past several times. The storagecontrol unit 124 generates a position map 323 obtained by averaging suchis position maps and updates the entry according to the position map323. Also, the storage control unit 124 may update the 3D shapeinformation or the image information in the same manner.

<2.2. Provided Information Display Function>

Next, the provided information display function will be described. Theinformation processing system 100 uses the correspondence relation DB,the real object DB and the display object DB updated by the DB updatefunction and performs a provided information display process ofdisplaying the provided information.

(Provided Information Display Process)

FIG. 13 is a flowchart illustrating an exemplary flow of providedinformation display processes performed in the information processingsystem 100 according to an embodiment.

As illustrated in FIG. 13, first, in step S502, the informationprocessing system 100 performs the real object recognition process. Thisprocess is the same as described above with reference to FIG. 7.

Next, in step S504, the recognition unit 122 determines whether a newreal object is added. When it is determined that the new real object isnot added (NO in S504), the process returns to step S502 again.

When it is determined that the new real object is added (YES in S504),in step S506, the information processing system 100 performs the realobject identification process of the added real object. This process isthe same as described above with reference to FIG. 8. Note that theinformation processing system 100 may perform the real object DB updateprocess described above with reference to FIG. 11 together.

Next, in step S508, the recognition unit 122 performs a display objectidentification process. The display object identification process is aprocess of identifying the display object that is currently displayed.For example, the recognition unit 122 acquires information of thedisplay object displayed on the display surface based on informationoutput by the output unit 130 and identifies the display object withreference to the display object DB.

Next, in step S510, the storage control unit 124 determines whether atleast any of the real object identified in step S506 and the displayobject identified in step S508 is registered in the correspondencerelation DB.

When it is determined that the object is not registered (NO in S510),the process ends. When it is determined that the object is registered(YES in S510), in step S512, the information processing system 100displays the provided information with reference to the correspondencerelation DB. This process will be described below in detail withreference to FIG. 14.

FIG. 14 is a flowchart illustrating an exemplary flow of providedinformation display processes performed in the information processingsystem 100 according to an embodiment.

As illustrated in FIG. 14, first, in step S602, the storage control unit124 acquires the associated entry from the local correspondence relationDB. For example, the storage control unit 124 acquires an entryincluding at least any of the real object identified in step S506 andthe display object identified in step S508 from the correspondencerelation DB. Within the acquired entry, using the entry on whichprocesses of steps S604 to S608 are not performed as a target, processesafter this step are performed for each target entry.

Next, in step S604, it is determined whether the provided informationbased on the same entry is already displayed. When it is determined thatthe information is displayed (YES in S604), the process advances to thefollowing step S610. When it is determined that the information is notdisplayed (NO in S604), in step S606, the display control unit 126determines whether a degree of association of the target entry exceeds athreshold value.

When it is determined that a degree of association does not exceed athreshold value (NO in S606), the process advances to the following stepS610. When it is determined that a degree of association exceeds athreshold value (YES in S606), in step S608, the display control unit126 displays the provided information. For example, when a part of thecombination of the correspondence relation information is recognized andthe other part thereof is not recognized, the display control unit 126may display information on the display object or the real object whichis not recognized as the provided information. In this case, the displaycontrol unit 126 may also display the provided information based onarrangement information. For example, when the real object is recognizedand the associated display object is not recognized, the display objectis displayed in a relative direction, distance, orientation, and sizeusing the recognized real object as a reference. Therefore, for example,when the user places only the real object on the table 140A, anapplication that is often used together with the real object can beactivated in a mode that is often used. In addition, the display controlunit 126 may display advertisement information of the display object orthe real object as the provided information.

Here, the display control unit 126 may preferentially display theprovided information based on the correspondence relation information ofa combination having a high degree of similarity, that is, when theobject has been recognized many times or for a long time by therecognition unit 122. Therefore, for example, when the user places onlythe real object on the table 140A, an application that is often usedtogether with the real object can be preferentially activated. Inaddition, when the real object or the display object is recognized, thedisplay control unit 126 may apply a specified time lag and display theprovided information. Therefore, for example, when the user places onlythe real object on the table 140A, an application that is often usedtogether with the real object can be activated in connection with atiming at which use is started.

Note that, the display control unit 126 may also display the providedinformation according to the user's biometric information recognized bythe recognition unit 122. For example, when it is recognized that theuser's body temperature is high, the display control unit 126 maydisplay enlarged characters in order for the user who is estimated to bein an excited state. In this manner, the user can receive the provisionof information according to her or his own state. In addition, thedisplay control unit 126 may display the provided information accordingto a device near the user recognized by the recognition unit 122. Forexample, when the user does not wear a wristwatch, the display controlunit 126 may display a clock application. In this manner, the user canreceive the provision of information according to the device near her orhim.

The process in step S608 has been described above. Hereinafter,description will return to the flowchart.

Next, in step S610, the storage control unit 124 determines whether allacquired entries are processed. When it is determined that the entriesare not processed (NO in S610), the process returns to step S604. Whenit is determined that the entries are processed (YES in S610), in stepS612, the storage control unit 124 acquires an associated entry from thecloud correspondence relation DB.

Next, in step S614, the display control unit 126 displays the providedinformation based on the entry acquired from the cloud correspondencerelation DB. The processing content is the same as in step S608. Whilethe local correspondence relation DB learns a use method of the user whouses the information processing system 100, the cloud correspondencerelation DB refers to integrated knowledge including other users. Forthis reason, the display control unit 126 may automatically activate,for example, an application in step 608, or may perform, for example,display that suggests an activation of an application in asemitransparent manner in this step. Therefore, the display control unit126 may perform activation when reception is performed within apredetermined time, or may withdraw when reception is not performed.

3. SPECIFIC EXAMPLES

Next, specific examples of functions implemented by the informationprocessing system 100 according to an embodiment will be described.

(Specific Examples of DB)

First, specific examples of the DB stored in the storage unit 150 areshown in FIGS. 15 to 17, and Tables 5 and 6.

FIG. 15 is a diagram illustrating exemplary information stored in thedisplay object DB according to an embodiment. As illustrated in FIG. 15,a display object of an entry D1 is a television application. A displayobject of an entry D2 is a news application. A display object of anentry D3 is a timer application. A display object of an entry D4 is atimer application. A display object of an entry D5 is a radioapplication. A display object of an entry D6 is a calendar application.A display object of an entry D7 is a leaflet application. A displayobject of an entry D8 is a recipe application.

FIGS. 16 and 17 are diagrams illustrating exemplary information storedin the real object DB according to an embodiment. As illustrated in FIG.16, a real object of an entry R1 is a cup of noodles. A real object ofan entry R2 is a cover of the cup of noodles. A real object of an entryR3 is a coffee cup that a husband uses in the morning. A real object ofan entry R4 is a coffee cup that a wife uses in the morning. Asillustrated in FIG. 17, a real object of an entry R5 is a western meal.A real object of an entry R6 is a teapot. A real object of an entry R7is a teacup. A real object of an entry R8 is flowers. A real object ofan entry R9 is cutlery. Also, in FIGS. 16 and 17, a pattern of 3D shapeinformation indicates a height from the table 140A.

Table 5 is a table that shows exemplary information stored in the localcorrespondence relation DB according to an embodiment. Table 6 is atable that shows exemplary information stored in the cloudcorrespondence relation DB according to an embodiment. As an example,the arrangement information exemplifies a direction of the displayobject seen from the real object. In addition, in order to facilitateunderstanding, content of identification information of the displayobject and identification information of the real object is described inparentheses.

TABLE 5 Identification Identification Identification informationinformation of information Arrangement Degree of of entry display objectof real object information association LE1 D3 R1 Bottom 20 (Timer) (Cupof right noodles) LE2 D1 R3 Left 10 (Television) (Coffee cup thathusband uses in morning) LE3 D4 R6 Right 5 (Timer) (Teapot) LE4 D4 R7Top 8 (Timer) (Teacup) LE5 D7 R4 Left 10 (Leaflet) (Coffee cup that wifeuses in morning) LE6 D8 R5 Top 6 (Recipe) (Western meal)

TABLE 6 Identification Identification Identification informationinformation of information Arrangement Degree of of entry display objectof real object information association CE1 D2 R1 Right 10 (News) (Cup ofnoodles) CE2 D6 R3 Left 80 (Calendar) (Coffee cup that husband uses inmorning) CE3 D4 R6 Left 10 (Radio) (Teapot) CE4 D8 RS Top 20 (Recipe)(Western meal) CE5 D8 R9 Right 30 (Recipe) (Cutlery)

(Specific Examples in Which a Real Object is Identified)

Next, specific examples in which a real object is identified will bedescribed with reference to FIGS. 18 and 19.

FIG. 18 is a diagram for describing an exemplary process of identifyingthe real object according to an embodiment. A real object 401 is a cupof noodles, and is assigned the identification information R1 asillustrated in FIG. 16. A real object 402 is a cover of the cup ofnoodles and is assigned the identification information R2 as illustratedin FIG. 16. A real object 403 is a vessel that resembles the cup ofnoodles and is not registered in the real object DB.

Here, an example in which the real objects 401 to 403 are placed at asimilar position on the table 140A and the real object of the entry R1is identified is considered. The real objects A to C of the exampledescribed with reference to Equation 2 and Table 4 are replaced with thereal objects 401 to 403 and the registered real object is replaced withthe real object of the entry R1 for this example. As shown in Table 4,in the real objects 401 to 403, there is no difference in theconcordance rate M_(i) of the captured image due to a similarappearance, but there is a significant difference in the concordancerate M_(d) of the depth information. For this reason, the storagecontrol unit 124 can distinguish and identify these real objects.

FIG. 19 is a diagram for describing an exemplary process of identifyingthe real object according to an embodiment. FIG. 19 illustrates examplesof UIs in a time period of morning.

As indicated by the reference numeral 411, first, the husband is sittingin a seat and places a coffee cup 421 on the table 140A. The coffee cup421 is registered in the real object DB as the entry R3 as illustratedin FIG. 16, and is registered in the correspondence relation. DBassociated with the television application in the entry LE2 as shown inTable 5. Therefore, the information processing system 100 refers to theentry LE2, and activates a television application 422 to the left of thecoffee cup 421 as indicated by the reference numeral 412.

Next, as indicated by the reference numeral 413, the wife is sitting ina seat and places a coffee cup 423 on the table 140A. The coffee cup 423is registered in the real object DB as the entry R4 as illustrated inFIG. 16 and is registered in the correspondence relation DB associatedwith the leaflet application in the entry LE5 as shown in Table 5.Therefore, the information processing system 100 refers to the entry LE5and activates a leaflet application 424 to the left of the coffee cup423 as indicated by the reference numeral 414.

In this manner, even real objects having substantially the same imageinformation and 3D shape information can be distinguished by theinformation processing system 100 using the position map, and can beassociated with different display objects. Further, in order to displaythe display object at a relative position using the real object as areference, the information processing system 100 can activate anapplication in front of the user's eyes, for example, when the user putsa cup of coffee down with her or his right hand.

Here, it can be understood that it is difficult for a camera configuredto capture an image from above the table 140A to capture an image of aface of the user and recognize the user. However, as exemplified in FIG.19, the information processing system 100 can practically identify theuser such as the husband or the wife using the position map. Inaddition, when the time period is different even if the user is thesame, the information processing system 100 can activate a differentapplication when the same coffee cup is placed at the same position. Inthis manner, when a use method is different even if the real object isthe same, the information processing system 100 can display a differentdisplay object appropriate for the use method.

(Specific Examples of Local Correspondence Relation DB)

Next, specific examples in which an entry is registered in the localcorrespondence relation DB will be described with reference to FIGS. 20and 21.

FIG. 20 is a diagram for describing an exemplary process of the localcorrespondence relation DB according to an embodiment. An example inwhich the display object and the real object are initially associatedwill be described with reference to FIG. 20.

As indicated by the reference numeral 431, the user watches a televisionapplication 441 on the table 140A. Next, as indicated by the referencenumeral 432, the user places a coffee cup 442 to the right of thetelevision application 441. In this case, the information processingsystem 100 performs the correspondence relation DB update process andregisters the entry LE2 shown in Table 5. Also, the televisionapplication 441 is positioned to the left of the coffee cup 442. Whenthe state indicated by the reference numeral 432 is reproduced aplurality of times, the degree of association of the entry LE2increments and becomes equal to or greater than the threshold value.Therefore, as indicated by the reference numeral 433, when the userplaces the coffee cup 442 on the table 140A, the information processingsystem 100 automatically activates the television application 442 to theleft of the coffee cup 442, as indicated by the reference numeral 434.

FIG. 21 is a diagram tier describing an exemplary process of the localcorrespondence relation DB according to an embodiment. An example inwhich the display object and the real object are initially associatedwill be described with reference to FIG. 21.

As indicated by the reference numeral 451, the user places a cup ofnoodles 461 on the table 140A and touches a launcher 462. Therefore, asindicated by the reference numeral 452, an application list 463 isdisplayed, and the user drags a timer application 464 within the list tothe lower right of the cup of noodles 461 and activates it. Then, theuser closes the application list 463 as indicated by the referencenumeral 453, and uses the timer application 464 as indicated by thereference numeral 454. In this case, the information processing system100 performs the correspondence relation DB update process and registersthe entry LE1 as shown in Table 5. Therefore, when the cup of noodles isplaced, the information processing system 100 automatically activatesthe timer application to the lower right thereof. On the other hand,when the user activates the timer application, the informationprocessing system 100 may display information for suggesting that thecup of noodles should be placed to the upper left thereof oradvertisement information for another cup of noodles.

(Specific Examples of Cloud Correspondence Relation DB)

Next, specific examples in which provided information is displayed withreference to the cloud correspondence relation DB will be described withreference to FIGS. 22 to 25.

FIG. 22 is a diagram for describing an exemplary process of the cloudcorrespondence relation DB according to an embodiment. An example inwhich provided information is displayed based on the cloudcorrespondence relation DB will be described with reference to FIG. 22.

As indicated by the reference numeral 471, the user places a cup ofnoodles 481 on the table 140A. Therefore, the information processingsystem 100 refers to the entry LE1 shown in Table and activates a timerapplication 482 to the lower right of the cup of noodles 481 asindicated the by reference numeral 472. Then, the information processingsystem 100 refers to the entry CE1 registered in the cloudcorrespondence relation DB shown in Table 6 and displays asemitransparent display object 483 for suggesting activation of the newsapplication as indicated by the reference numeral 473. As indicated bythe reference numeral 473, when the user touches the semitransparentdisplay object 483 and accepts the suggestion, the informationprocessing system 100 displays a news application 484 as indicated bythe reference numeral 474. Further, the information processing system100 may register the correspondence relation information associated withthe cup of noodles 481 and the news application 484 in the localcorrespondence relation DB.

FIG. 23 is a diagram for describing an exemplary process of the cloudcorrespondence relation DB according to an embodiment. An example inwhich provided information is displayed based on the cloudcorrespondence relation DB will be described with reference to FIG. 23.

As indicated by the reference numeral 491, the user places a coffee cup501 on the table 140A and watches a television application 502. Then,the information processing system 100 refers to the entry CE2 registeredin the cloud correspondence relation DB shown in Table 6, and displays asemitransparent display object 503 for suggesting activation of acalendar application as indicated by the reference numeral 492. Asindicated by the reference numeral 493, when the user touches thesemitransparent display object 503 and accepts the suggestion, theinformation processing system 100 displays a calendar application 504 asindicated by the reference numeral 494. Further, the informationprocessing system 100 may register the correspondence relationinformation associated with the coffee cup 501 and the calendarapplication 504 in the local correspondence relation DB.

FIG. 24 is a diagram for describing an exemplary process of the cloudcorrespondence relation DB according to an embodiment. An example inwhich provided information is displayed based on the cloudcorrespondence relation DB will be described with reference to FIG. 24.

As indicated by the reference numeral 511, the user places a teacup 521and a teapot 522 on the table 140A. Then, the information processingsystem 100 refers to the entries LE3 and LE4 registered in the localcorrespondence relation DB shown in Table 5 and activates a timerapplication 523 as indicated by the reference numeral 512. Therefore,the information processing system 100 refers to the entry CE3 registeredin the cloud correspondence relation DB shown in Table 6 and displaysthe semitransparent display object 524 for suggesting activation of aradio application as indicated by the reference numeral 513. Asindicated by the reference numeral 513, when the user touches thesemitransparent display object 524 and accepts the suggestion, theinformation processing system 100 displays a radio application 525 asindicated by the reference numeral 514. Further, the informationprocessing system 100 may register the correspondence relationinformation associated with the teacup 521 and/or the teapot 522 and theradio application 525 in the local correspondence relation DB.

FIG. 25 is a diagram for describing an exemplary process of the cloudcorrespondence relation DB according to an embodiment. An example inwhich provided information is displayed based on the cloudcorrespondence relation DR will be described with reference to FIG. 25.

As indicated by the reference numeral 531, the user places a westernmeal 541 on the table 140A decorated with flowers 542. Then, theinformation processing system 100 refers to the entry LE6 shown in Table5 and activates a recipe application 543 as indicated by the referencenumeral 532. Also, as shown in Table 6, even in the cloud correspondencerelation DB, the same entry CE4 is registered. However, since theinformation processing system 100 has already activated the recipeapplication 543, suggestion information based on the entry CE4 is notdisplayed. On the other hand, the information processing system 100refers to the entry CE5 registered in the cloud correspondence relationDB shown in Table 6 and displays a display object 544 of recommendingthe cutlery as indicated by the reference numeral 533. As indicated bythe reference numeral 533, when the user becomes interested in thesuggestion and touches it, the information processing system 100activates a web browser 545 displaying a purchase site of the cutlery asindicated by the reference numeral 534.

4. HARDWARE CONFIGURATION EXAMPLE

Finally, a hardware configuration of an information processing deviceaccording to an embodiment will he described with reference to FIG. 26.FIG. 26 is a block diagram illustrating an exemplary hardwareconfiguration of an information processing device according to anembodiment. Also, an information processing device 900 illustrated inFIG. 26 may implement, for example, the information processing system100 illustrated in FIG. 4 and the input and output device 200 and theserver 300 illustrated in FIG. 5. An information process by theinformation processing system 100 according to an embodiment isimplemented by cooperation of software and hardware to be describedbelow.

As illustrated in FIG. 26, the information processing device 900includes a central processing unit (CPU) 01, a read only memory (ROM)902, a random access memory (RAM) 903 and a host bus 904 a. In addition,the information processing device 900 includes a bridge 904, an externalbus 904 b, an interface 905, an input device 906 an output device 907, astorage device 908, a drive 909, a connection port 911 and acommunication device 913. The information processing device 900 mayinclude a processing circuit such as a DSP or an ASIC instead of or inaddition to the CPU 901.

The CPU 901 serves as an arithmetic processing unit and a control deviceand controls overall operations in the information processing device 900according to various programs. In addition, the CPU 901 may be amicroprocessor. The ROM 902 stores a program, a calculation parameter orthe like that is used by the CPU 901. The RAM 903 temporarily stores aprogram that is used when the CPU 901 is executed, and a parameter thatis appropriately changed in the execution. The CPU 901 may form, forexample, the recognition unit 122, the storage control unit 124 and thedisplay control unit 126 illustrated in FIGS. 4 and 5.

The CPU 901, the ROM 902 and the RAM 903 are connected to one another bythe host bus 904 a including a CPU bus. The host bus 904 a is connectedto the external bus 904 b such as a Peripheral ComponentInterconnect/Interface (PCI) bus through the bridge 904. Also, it is notnecessary to configure the host bus 904 a, the bridge 904 and theexternal bus 904 b separately, but such functions may be implemented inone bus.

The input device 906 is implemented by a device to which information isinput by the user, for example, a mouse, a keyboard, a touch panel, abutton, a microphone, a switch and a lever. In addition, the inputdevice 906 may be, for example, a remote control device using infraredrays or other radio waves, or may be an external connection device suchas a mobile phone or a PDA corresponding to a manipulation of theinformation processing device 900. In addition, the input device 906 maybe a visible light camera, an infrared camera, a stereo camera, adistance measuring sensor using a time of flight method, a distancemeasuring sensor using a structured light method, or a biometric sensor.Further, the input device 906 may include an input control circuit thatgenerates an input signal based on information input by the user using,for example, the above input device, and outputs the signal to the CPU901. The user of the information processing device 900 may input varioustypes of data or instruct a process operation with respect to theinformation processing device 900 by manipulating the input device 906.The input device 906 may form, for example, the input unit 110illustrated in FIGS. 4 and 5.

The output device 907 is formed as a device capable of visually oraudibly notifying the user of the acquired information. Such a deviceincludes a display device such as a projector, a CRT display device, aliquid crystal display device, a plasma display device, an EL displaydevice and a lamp, an audio output device such as a speaker and aheadphone, a printer device or the like. The output device 907 outputsresults obtained by various processes performed by, for example, theinformation processing device 900. Specifically, the display devicevisually displays results obtained by various processes performed by theinformation processing device 900 in various forms such as text, animage, a table, or a graph. On the other hand, the audio output deviceconverts an audio signal of reproduced audio data or acoustic data intoan analog signal and audibly outputs the result. The display device andthe audio output device may form, for example, the output unit 130illustrated in FIGS. 4 and 5.

The storage device 908 is a device for data storage formed as an exampleof the storage unit of the information processing device 900. Thestorage device 908 is implemented by, for example, a magnetic storagedevice such as an HDD, a semiconductor storage device, an opticalstorage device or a magneto optical storage device. The storage device908 may include a storage medium, a recording device configured torecord data in the storage medium, a reading device configured to readdata from the storage medium, and a deleting device configured to deletedata recorded in the storage medium. The storage device 908 stores aprogram and various types of data executed by the CPU 901 and varioustypes of data acquired from the outside. The storage device 908 mayform, for example, the storage unit 150 illustrated in FIGS. 4 and 5.

The drive 909 is a reader/writer for the storage medium and is built inthe information processing device 900 or externally attached. The drive909 reads information recorded in a removable storage medium such as aninstalled magnetic disk, optical disc, magneto optical disc, orsemiconductor memory, and outputs the information to the RAM 903. Inaddition, the drive 909 can write information in the removable storagemedium.

The connection port 911 is an interface connected to an external deviceand is connection port of the external device capable of transmittingdata through, for example, Universal Serial Bus (USB).

The communication device 913 is a communication interface formed in acommunication device for connection with, for example, a network 920.The communication device 913 is a communication card for, for example, awired or wireless local area network (LAN), Long Term Evolution (LIE),Bluetooth (registered trademark), or wireless USB (WUSB). In addition,the communication device 913 may be a router for optical communication,a router for Asymmetric Digital Subscriber Line (ADSL), or a modem forvarious types of communication. The communication device 913 cantransmit and receive a signal or the like in compliant with apredetermined protocol, for example, TCP/IP, through, for example, theInternet, or with other communication devices. The communication device913 may form, for example, the communication unit 160 and thecommunication unit 170 illustrated in FIG. 5.

Also, the network 920 is a wired or wireless transmission path ofinformation that is transmitted from a device connected to the network920. For example, the network 920 may include a public network such asthe Internet, a telephone line network, and a satellite communicationnetwork, various types of local area networks (LANs) including Ethernet(registered trademark), and a wide area network (WAN). In addition, thenetwork 920 may include a dedicated line network such as InternetProtocol-Virtual Private Network (IP-VPN).

The exemplary hardware configuration in which functions of theinformation processing device 900 according to an embodiment can beimplemented has been illustrated above. The respective componentsdescribed above may be implemented using a general purpose member or maybe implemented by hardware that is specialized for functions ofrespective components. Therefore, it is possible to appropriately changea hardware configuration to be used according to a technical level whenan embodiment is implemented.

Note that a computer program for implementing the above-describedrespective functions of the information processing device 900 accordingto an embodiment can be prepared and installed in a PC or the like. Inaddition, it is possible to provide a computer readable recording mediumin which such a computer program is stored. The recording mediumincludes, for example, a magnetic disk, an optical disc, a magnetooptical disc and a flash memory. In addition, the computer program maybe delivered through, for example, a network, without using therecording medium.

5. SUMMARY

An embodiment of the present disclosure has been described above indetail with reference to FIGS. 1 to 26. As described above, theinformation processing system 100 according to an embodiment acquiresinformation indicating a correspondence relation between the displayobject and the real object from the storage unit, and displays providedinformation corresponding to at least any of the display object and thereal object recognized on the object surface on which the display objectis displayed based on information indicating the correspondencerelation. Therefore, the user can receive the provision of informationof the real object corresponding to, for example, the display objectthat is displayed, and receive the provision of information of thedisplay object corresponding to the real object placed on the displaysurface. For example, when the user places only the real object on thedisplay surface, since an application having a high degree ofassociation is automatically activated, it is possible to omit anactivation step. In this manner, the user convenience may increase.

In addition, the information processing system 100 identifies the realobject according to a captured image, 3D shape information, and aposition map of the real object. Therefore, the information processingsystem 100 can identify a real object having no texture, real objectshaving the same appearance and different heights, and real objectshaving the same appearance and height and different use positions. Inaddition, the information processing system 100 can practicallyimplement user recognition, which is difficult in image recognitionusing a camera configured to capture an image from above the user, usingthe position map. That is, when a use method is different even if thereal object is the same, the information processing system 100 candisplay different provided information. In addition, since theinformation processing system 100 identifies the real object in acombination of the captured image, the 3D shape information, and theposition map, recognition accuracy of the real object may increasecompared to a recognition method using only the captured image. Inaddition, the information processing system 100 controls weightcoefficients of Equation 1 used for recognition, and thus can optimize ause ratio of the captured image, the 3D shape information, and theposition map according to a use environment. In the technique disclosedin Patent Literature 1 described above, since only an image is used foridentification, it is difficult to identify a real object having notexture, real objects having the same appearance and different heights,and real objects having the same appearance and height and different usepositions.

In addition, the information processing system 100 displays the providedinformation according to information indicating a relative arrangementincluded in the correspondence relation information. Therefore, theinformation processing system 100 can display the provided informationin an appropriate arrangement seen from the real object, and display theprovided information in an appropriate arrangement seen from the displayobject.

In addition, the information processing system 100 stores an entryassociated with the combination of the recognized display object andreal object in the correspondence relation DB. Therefore, even when theuser is not aware of the association between the real object and thedisplay object, the information processing system 100 can learn a usagestate of the user and present information with high accuracy. In PatentLiterature 1 described above, only presentation of informationpreviously associated with an image is disclosed, and dynamic learningand updating of the correspondence relation is not disclosed.

In addition, the information processing system 100 may display providedinformation of another display object or another real object associatedwith at least any of the recognized display object and real object. Forexample, when the information processing system 100 refers to the cloudcorrespondence relation DB, it is possible to suggest an applicationthat the user does not know, and a new product that matches a usagestate of the table 140A. Further, when a third party registersadvertisement information in the cloud correspondence relation DB, it ispossible to recommend an application, a product or the like for theuser, and perform customer development.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, in the embodiment, the example in which the informationprocessing system 100 is used by a fixed user such as the husband or thewife was mainly described, but the present technology is not limitedthereto. For example, the information processing system 100 may beinstalled in a restaurant or a cafe and used by a plurality ofunspecified customers. In this case, for example, the informationprocessing system 100 may recommend a meal menu according to anapplication activated by a customer and display information according toa drink that the customer drinks.

In addition, respective devices described in this specification may beimplemented by a single device, or a part or all thereof may beimplemented by separate devices. For example, the recognition unit 122,the storage control unit 124 and the display control unit 126 may beprovided in separate devices.

In addition, in the embodiment, the example in which the recognitionunit 122 recognizes the display object and the real object has beendescribed, but the present technology is not limited thereto. Forexample, a first recognition unit configured to recognize the displayobject and a second recognition unit configured to recognize the realobject may be provided. Further, the first recognition unit and thesecond recognition unit may be provided in separate devices.

Note that it is not necessary for the processing described in thisspecification with reference to the flowchart to be executed in theorder shown in the flowchart. Some processing steps may be performed inparallel. Further, some of additional steps can be adopted, or someprocessing steps can be omitted. In addition, the process of theflowchart may be repeatedly performed or the process may end at any timeaccording to an end interrupt.

In addition, the effects described in the present specification aremerely illustrative and demonstrative, and not limitative. In otherwords, the technology according to the present disclosure can exhibitother effects that are evident to those skilled in the art along with orinstead of the effects based on the present specification.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus, including: circuitry configuredto

control output of a first content according to first contentinformation,

determine information related to a real object which is different from amanipulation body used for content selection or object manipulation by auser, and

control output of a second content, which is different from the firstcontent, according to second content information, wherein the secondcontent information is based on the first content information and thedetermined information related to the real object.

(2) The information processing apparatus of (1), wherein the output ofthe first content includes a first display object.

(3) The information processing apparatus of (1) or (2), wherein thefirst content information includes a first display location of the firstdisplay object on a display surface.

(4) The information processing apparatus of any of (1) to (3), whereinthe first content information includes identification information of thefirst display object.

(5) The information processing apparatus of any of (1) to (4), whereinthe output of the second content includes a second display object.

(6) The information processing apparatus of any of (1) to (5), whereinthe second content information includes a second display location of thesecond display object on the display surface.

(7) The information processing apparatus of any of (1) to (6), whereinthe circuitry initiates display of the second display object when theinformation related to the real object indicates a spatial relationshipbetween the real object and one or more components of the circuitry.

(8) The information processing apparatus of any of (1) to (7), whereinthe spatial relationship with one or more components of the circuitry isa state in which the real object is worn or held by the user within apredetermined distance from one or more components of the circuitry.

(9) The information processing apparatus of any of (1) to (8), whereinthe spatial relationship with one or more components of the circuitry isa state of being within a threshold distance from the display surface.

(10) The information processing apparatus of any of (1) to (9), whereinthe spatial relationship with one or more components of the circuitry isa state of being in contact with the surface of the display area.

(11) The information processing apparatus of any of (1) to (10), whereinthe information related to the real object is based on a captured imageof the real object.

(12) The information processing apparatus of any of (1) to (11), whereinthe captured image of the real object indicates a location of the realobject with respect to one or more components of the circuitry.

(13) The information processing apparatus of any of (1) to (12), whereinthe captured image of the real object indicates a shape of the realobject.

(14) The information processing apparatus of any of (1) to (13), whereinthe information related to the real object is based on a captured imageof the real object,

(15) The information processing apparatus of any of (1) to (14), whereinthe captured image of the real object indicates a location of the realobject with respect to the display surface.

(16) The information processing apparatus of any of (1) to (15), whereinthe circuitry initiates display of the second display object when thelocation of the real object is within a threshold distance from thedisplay surface.

(17) The information processing apparatus of any of (1) to (16), whereinthe circuitry initiates display of the second display object assuggested content with limited functional or display information, andupon determination by the user, the second display object is output withfull functional and display information.

(18) The information processing apparatus of any of (1) to (17), whereinupon the determination by the user, the second display object and seconddisplay location are registered in association with a combination of thefirst content information and the determined information related to thereal object, such that the second display object is output at thedetermined second display location with full functional and displayinformation when the combination of the first content information andthe determined information related to the real object is detected.

(19) The information processing apparatus of any of (1) to (18), whereinthe second content information is further based on at least one ofinformation of time and information of a user profile.

(20) The information processing apparatus of any of (1) to (19), whereinthe second content information is further based on a user's biometricinformation.

(21) An information processing method including:

outputting a first content according to first content information;

determining information related to a real object which is different froma manipulation body used for content selection or object manipulation bya user; and

outputting a second content, which is different from the first content,according to second content information, wherein the second contentinformation is based on the first content information and the determinedinformation related to the real object.

(22) A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute a method, the method including:

outputting a first content according to first content information;

determining information related to a real object which is different froma manipulation body used for content selection or object manipulation bya user; and

outputting a second content, which is different from the first content,according to second content information, wherein the second contentinformation is based on the first content information and the determinedinformation related to the real object.

(23)

An information processing system including:

a display control unit configured to cause provided informationcorresponding to at least any of a recognized display object and realobject to be displayed on a display unit based on a recognition resultof the display object and the real object on an object surface on whichthe display object is displayed.

(24)

The information processing system according to (23), wherein the displaycontrol unit causes the provided information to be displayed based on aposition of the recognized display object or real object.

(25)

The information processing system according to (24), wherein the displaycontrol unit causes the provided information to be displayed based onpositions of the recognized display object and real object.

(26)

The information processing system according to any one of (23) to (25),wherein the provided information is generated based on a position of therecognized display object or a position of the recognized real object,

(27)

The information processing system according to (26), wherein theprovided information is determined based on a position of the recognizeddisplay object and a position of the recognized real object.

(28)

The information processing system according to (23), wherein the displaycontrol unit causes the provided information to be displayed based oninformation indicating a correspondence relation between the displayobject and the real object.

(29)

The information processing system according to (28), wherein theinformation indicating the correspondence relation includes informationindicating a relative arrangement between the real object and thedisplay object, and wherein the display control unit causes the providedinformation to be displayed based on information indicating the relativearrangement.

(30)

The information processing system according to (28) or (29), furtherincluding: a storage control unit configured to acquire the informationindicating the correspondence relation from a storage unit.

(31)

The information processing system according to (30), wherein the storagecontrol unit identifies the real object according to a captured image,information indicating a 3D shape, or information indicating a positionat the object surface of the real object.

(32)

The information processing system according to (31), wherein the storagecontrol unit identifies the real object according to informationindicating a time period in which the real object is detected.

(33)

The information processing system according to any one of (30) to (32),wherein the storage control unit causes the information indicating thecorrespondence relation associated with a combination of the recognizeddisplay object and real object to be stored in the storage unit.

(34)

The information processing system according to any one of (28) to (33),wherein the provided information is information of the display objectassociated with the recognized real object in information indicating thecorrespondence relation.

(35)

The information processing system according to any one of (28) to (34),wherein the provided information is information of the display objectassociated with the recognized display object in information indicatingthe correspondence relation.

(36)

The information processing system according to (35), wherein the displaycontrol unit causes the provided information to be preferentiallydisplayed based on the information indicating the correspondencerelation of a combination of the display object and the real object thathave been recognized many times or for a long time.

(37)

The information processing system according to any one of (23) to (36),wherein, when the real object or the display object is recognized, thedisplay control unit causes the provided information to be displayedafter a specified time lag.

(38)

The information processing system according to any one of (23) to (37),wherein the display control unit causes advertisement informationassociated with at least any of the recognized display object and realobject to be displayed.

(39)

The information processing system according to any one of (23) to (38),wherein the display control unit causes the provided information to bedisplayed according to a user's recognized biometric information.

(40)

The information processing system according to any one of (23) to (17),wherein the display control unit causes the provided information to bedisplayed according to a recognized device near the user.

(41)

An information processing method including:

controlling, by a processor, provided information corresponding to atleast any of a recognized display object and real object to be displayedon a display unit based on a recognition result of the display objectand the real object on an object surface on which the display object isdisplayed.

(42)

A program causing a computer to function as

a display control unit configured to cause provided informationcorresponding to at least any of a recognized display object and realobject to be displayed on a display unit based on a recognition resultof the display object and the real object on an object surface on whichthe display object is displayed.

(43)

A server including:

a display control unit configured to generate a display control signalfor causing provided information corresponding to at least any of arecognized display object or real object to be displayed on a displayunit based on a recognition result of the display object and the realobject on an object surface on which the display object is displayed.

(44)

The information processing system according to any one of (28) to (36),wherein the display control unit causes the provided information to bedisplayed based on information indicating the correspondence relation inanother information processing system.

(45)

The information processing system according to (44), wherein the displaycontrol unit causes the provided information to be displayed based oninformation indicating the correspondence relation of another user whois similar to a recognized user.

REFERENCE SIGNS LIST

100 information processing system

110 input unit

120 control unit

122 recognition unit

124 storage control unit

126 display control unit

130 output unit

140 table

150 storage unit

160 communication unit

170 communication unit

200 input and output device

300 server

1. An information processing apparatus, comprising: circuitry configuredto determine content information related to an object in real space,control display of content at an arrangement position based on thecontent information, wherein the content information includesarrangement information describing the arrangement position of thedisplayed content as a relative position with respect to the object inthe real space, and update the arrangement position based on a userinput.
 2. The information processing apparatus according to claim 1,wherein when the arrangement information is initially not registered,the circuitry controls display of the content at a predeterminedposition and registers the arrangement information based on the relativeposition with respect to the object in the real space.
 3. Theinformation processing apparatus according to claim 1, wherein when thearrangement information is already registered, the circuitry updates thearrangement information to increase a degree of association between theobject and the displayed content.
 4. The information processingapparatus according to claim 1, wherein when a position or orientationof the object is different than an initially determined position ororientation of the object, the circuitry employs a different arrangementfrom the arrangement position determined by the arrangement information.5. The information processing apparatus according to claim 1, whereinthe object comprises at least one of a real object or a display objectin the real space.
 6. The information processing apparatus according toclaim 5, wherein when the real object is recognized and the displayobject is not recognized, the content is displayed in a relativedirection, distance, orientation, and size using the recognized realobject as a reference.
 7. The information processing apparatus accordingto claim 5, wherein when the object comprises the display object in thereal space, the circuitry employs a different arrangement of the contentfrom an arrangement based on the registered arrangement informationaccording to a state of a process of the display object.
 8. Theinformation processing apparatus according to claim 7, wherein the stateof the process of the display object comprises at least one of inprogress or awaiting user input.
 9. The information processing apparatusaccording to claim 8, wherein when the state of the process of thedisplay object is awaiting the user input, the circuitry controlsdisplay of the content with a bigger scale than when the state of theprocess of the display object is not awaiting the user input.
 10. Theinformation processing apparatus according to claim 8, wherein when theuser input is in progress with respect to the display object, thecircuitry controls display of the content in a smaller scale than whenthe user input is not in progress.
 11. The information processingapparatus according to claim 1, wherein the arrangement informationincludes a relative direction, distance, orientation, and size of thedisplayed content.
 12. The information processing apparatus according toclaim 1, wherein the object is on a display surface in the real space.13. An information processing method, executed by at least oneprocessor, the method comprising: determining content informationrelated to an object in real space; displaying content at an arrangementposition based on the content information, wherein the contentinformation includes arrangement information describing the arrangementposition of the displayed content as a relative position with respect tothe object in the real space; and updating the arrangement positionbased on a user input.
 14. A non-transitory computer-readable mediumhaving embodied thereon a program, which when executed by a computercauses the computer to execute a method, the method comprising:determining content information related to an object in real space;displaying content at an arrangement position based on the contentinformation, wherein the content information includes arrangementinformation describing the arrangement position of the displayed contentas a relative position with respect to the object in the real space; andupdating the arrangement position based on a user input.